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An  attempt  was  made  to  validate  a  low  cost,  low  fidelity, 
computer  driven  flight  simulator.  The  validation  is  required  so 
that  the  simulator  can  be  used  as  a  criterion  task  to  see  whether 
we  can  predict  flight  performance  on  the  basis  of  performance  on 
other  tests  of  individual  cognitive  ability  like  attentlonal 
flexibility,  visual  representational  skill,  priority  setting  and 
planning.  The  simulator  was  based  on  instrument  flying  rather 
than  visual  contact  flying  and  incorporated  secondary  tasks  to 
further  tax  the  pilot's  capacity.  Simulator  performance  is  corre¬ 
lated  with  hours  of  flight  training,  the  best  correlations  coming 
from  conditions  which  impose  additional  task  demands,  .. 
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The  present  study  is  part  of  a  larger  project  aimed  at  the  analysis  of 
c  mplex  skills  Into  component  abilities  and  the  prediction  of  performance  on 
the  complex  skills  by  tests  of  the  component  abilities.  The  abilities  in  which 
we  are  most  interested  are  cognitive  abilities  (e.g.,  such  as  representational 
skill,  attentional  flexibility,  timesharing  ability,  executive  abilities).  We 
feel  that  these  abilities  are  very  important  in  many  tasks  which  superficially 
seem  to  be  primarily  motor  performance  tasks  (e.g.,  team  athletics,  piloting 
an  airplane,  driving  an  automobile,  or  operating  other  complex  machinery). 

Although  the  conceptual  problem  of  breaking  down  complex  tasks  into 
component  abilities  does  not  require  that  the  complex  tasks  be  practical 
ones,  obviously  the  spin  off  from  this  type  of  research  would  bo  much 
greater  if  important  practical  tasks  were  analyzed.  We  have  taken  the  task  oi 
piloting  an  airplane  as  an  example  of  an  important  practical  task.  This  report 
describes  an  attempt  to  validate  an  Inexpensive  flight  simulator.  If  "flying" 
the  simulator  proves  highly  related  to  piloting.  It  WU  make  sense  to  attempt 
to  predict  simulator  performance  on  component  tasks.  Besides  its  practical 
importance,  piloting  has  the  advantage  of  being  well-studied  and  of  having 
many  valid  simulations  reported.  It  also  has  well-defined  component  tasks  and 
seems  to  require  the  sorts  of  cognitive  component  abilities  we  wish  to  study. 

A  review  of  the  literature  on  flight  simulation  suggested  that  we  could 
use  our  laboratory  computer  to  simulate  pilot  performance  tasks  effectively 
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over  more  specialized  high  fidelity  alternatives  were  flexibility  for 
measurement  or  training  and  portability. 

Our  support  for  this  thinking  cooes  from  a  variety  of  sources.  It  is 
clear  from  the  huge  simulation  literature  that  well  thought  out  simulators  can 
be  a  training  aid.  For  example  see  Crawford  and  Brock,  1977  for  a  recent 
review  .  Povenmire  and  Roscoe,  1971  for  simulator  training  in  primary  flight 
training,  Reid  and  Cyrus,  1974  for  formation  flight  training,  Trolllp,  1977 
for  training  on  holding  patterns;  Ketchel,  1977  for  air  combat  training.  Many 
simulators  are  designed  to  look  and  act  as  much  as  possible  like  an  airplane. 

The  large  simulator  at  Williams  Air  Force  Base  is  perhaps  one  of  the  extremes 
tn  fidelity  in  on  the  ground  simulators.  However,  there  are  reports  of  good 
performance  of  small  computer  based  simulations  in  predicting  performance  on 
the  large  simulators  and  on  training.  The  people  at  the  Human  Resources 
Laboratory  at  Williams  report  that  the  small  computer  based  simulation  developed 
at  Arizona  State  when  informally  evaluated  seemed  to  be  a  good  predictor  of 
performance  on  the  large  simulator.  This  is  not  to  say  that  these  inexpensive 
simulations  could  take  the  place  of  the  large  simulators  in  specific  advanced 
training  or  familiarization  exercises  which  go  on  at  Williams  and  with  the 
Air  Combat  Maneuvering  Range  Program.  But  it  may  be  an  indication  that  the 
general  skills  needed  to  fly  airplanes  might  be  assessed  and  trained  by  using 
inexpensive  computer  simulations. 

There  has  been  a  clear  message  in  the  literature  that  in  training,  con¬ 
siderations  of  task  analysis  and  training  design  can  be  more  Important  than 
realism  (see  Caro,  1973).  Recent  examples  come  from  Trollip,  1977  and  Finnegen, 
1977.  They  shoved  that  training  on  holding  patterns  could  be  effectively 
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carried  out  using  computer  aided  instruction  which  did  not  emphasize  fidelity 
at  all.  They  emphasized  (as  did  Caro)  that  for  the  purposes  of  training, the 
computer  can  do  things  which  most  simulators  cannot  in  that  they  can  provide 
a  more  effective  learn'tv  environment  by  optimizing  feedback  and  the  like. 

The  flexibility  to  allow  maximization  of  learning  or  measurement  parameters 
can  be  very  important  as  long  as  the  essential  aspects  of  the  task  to  be 
learned  or  measured  have  been  successfully  abstracted  and  incorporated.  Our 
research  assumes  that  the  abstraction  of  component  abilities  is  possible  and 
can  be  done  practically.  The  ability  to  arrive  at  Rood  predictions  of  pilot 
performance  with  an  inexpensive,  unrealistic  model  is  one  test  of  this  assump¬ 
tion.  More  stringent  and  interesting  tests  will  be  to  see  whether  we  can 
abstract  and  Isolate  general  cognitive  components  useful  in  a  variety  of  other 
skills.  Support  for  useful  abstraction  of  general  cognitive  abilities  as 
components  of  piloting  comes  from  Taylor  and  Parker  (1959)  for  spatial  tents 
as  predictors  of  training  success*  Fleishman  and  Ornstein  (1960)  found  factors 
predicting  pilot  performance  were  control  precision,  spatial  orientation,  multi- 
limb  coordination,  response  orientation,  rate  control,  kinesthetic  discrimina¬ 
tion  and  Gopher  and  Kahneman  (1971)  found  that  differences  in  attention  predic¬ 
ted  advanced  placement  of  pilots. 

Early  in  our  research  we  tried  to  obtain  one  of  several  practical  working 
simulations.  These  were  either  not  available  or  not  compatible  with  our  equip¬ 
ment.  We  then  wrote  our  own  simulator  for  use  on  our  laboratory  computer  and 
the  studies  reported  were  done  on  this  version.  Due  to  computer  problems  on 
the  laboratory  computer  >ne  of  us  is  currently  implementing  a  second  version 
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Our  primary  target  for  this  research  is  the  more  sophisticated  pilot 
and  we  have  arbitrarily  designated  the  instrument  pilot  as  reaching  this 
level  of  sophistication.  Thus,  we  are  not  just  looking  for  someone  who  can 
drive  a  plane  but  someone  who  can  deal  with  other  complexities  at  the  same 
time,  as  would  any  pilot  flying  for  military  or  commercial  purposes.  Thus, 
we  have  made  our  simulation  task  very  demanding.  Maintaining  direction  is 
made  demanding  by  stiff  simulated  winds  and  by  simulated  turbulence.  A 
secondary  computation  task  is  also  required.  Subjects  are  given  a  speed  and 
a  distance  and  asked  to  compute  the  time  required  to  fly  the  distance.  In 
most  cases  where  a  decision  had  to  be  made  about  how  close  the  simulator 
should  be  to  the  real  thing,  we  decided  that  the  cognitive  task  should  be  simi¬ 
lar  but  the  motor  task  different.  Subjects  maintain  direction  with  two  push¬ 
buttons,  one  for  each  direction.  Instrument  and  control  dynamics  are  not 
those  commonly  found  in  planes.  Finally,  calculations  were  not  made  on  a 
flight  computer  and  the  procedures  were  different  than  is  commonly  used. 

Our  expectations  were  that  the  simulator  tasks  without  secondary  tasks 
would  not  be  as  related  to  flight  proficiency  as  would  the  simulator  tasks  with 
the  secondary  tasks.  In  general,  we  expected  the  more  demanding  tasks  to  be 
more  discriminating.  These  expectations  came  from  the  belief  that  tracking 
ability  alone  would  not  be  a  good  Indication  of  advanced  performance  on  a 
flight  task  whereas  the  ability  to  deal  with  task  complexity  would.  In  pre¬ 
liminary  work  we  found  that  people  seemed  to  rely  on  some  type  of  visualiza¬ 
tion  ability  even  for  direction  maintenance  on  the  simulator.  The  logic  of 
instrument  flying  and  earlier  research  (e.g.,  Taylor  and  Parker,  1959)  led  us 
to  believe  that  visualization  or  spatial  abilities  would  be  an  Important  factor 
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in  skilled  piloting.  Thus,  we  included  a  visual  interference  task  to  see 
yhether  we  could  lnpare  the  better  pilots.  An  auditory  interference  task 
was  expected  to  have  less  interfering  effect  on  the  good  pilots  than  the  visual 
interference  task. 

Method 

Subjects  were  recruited  from  the  Lane  Community  College  Flight  Techno¬ 
logy  Program.  The  trainees  ranged  from  pre-solo  to  instrument  instructor. 
Information  on  flight  training  experience  and  test  scores  was  gathered. 
Twenty-four  trainees  signed  up  to  participate,  eighteen  came  to  the  first 
session.  However,  because  of  numerous  computer  failures  all  of  the  subjects 
did  not  provide  data  for  all  conditions.  Failures  either  prevented  subjects 
from  finishing  a  condition  or  prevented  subjects  from  being  tested  at  all 
resulting  in  some  subjects  refusing  to  return  for  later  testing.  Subjects 
were  paid  $3.50  per  hour  and  were  promised  a  bonus  if  they  participated  in 
18  hours  of  testing  which  would  have  boosted  their  earning  to  $5. 00/hr. 

Work  on  the  simulator  was  scheduled  to  take  3  hours.  Because  of 
computer  problems  this  was  sometimes  broken  up  into  two  or  three  different 
sessions  on  different  dava. 

(see  Figure  1,  page  following)  > 

The  simulation  consists  of  three  instruments.  One  is  similar  to  a 
directional  Ryro,  one  is  similar  to  a  turn  indicator,  one  is  similar  to  a 
course  deviation  indicator  and  TO/FROM  indicator  of  a  VOR  receiver.  Note 
that  altitude  is  not  represented  in  the  version  of  the  simulator  reported 
here.  Suhjects  are  given  successive  courses  to  hold  in  no  wind,  no 
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turbulence  conditions  (Easy  Condition),  moderate  wind  (about  20/5  of  the  speed 
of  the  plane)  and  turbulence  conditions  (Moderate  Condition),  heavy  wind 
(about  50%  of  the  speed  of  the  plane)  and  turbulence  conditions  (Difficult 
Condition).  Simulated  wind  moved  the  position  of  the  plane  as  a  vector 
independent  of  the  heading  vector.  Simulated  turbulence  changed  the  heading. 
Each  individual  task  set  contained  4  courses  and  if  flown  perfectly  would  take 
about  8  minutes.  In  practice  each  took  about  10  minutes,  depending  on  skill 
level,  totaling  about  90  minutes  of  simulated  time  in  the  three  hour  session. 
Subjects  controlled  the  plane's  heading  by  pushing  one  of  two  buttons.  The 
longer  the  button  was  held  down,  the  steeper  the  turn.  To  come  out  of  a  turn 
the  opposite  button  had  to  be  pressed.  Subjects  were  to  fly  the  course  as 
directly  as  possible,  correcting  for  wind  and  turbulence  as  necessary. 

After  lengthy  instructions,  subjects  were  given  the  easy,  moderate 
and  difficult  conditions  in  that  order  without  other  tasks.  During  this  time 
subjects  could  ask  questions  and  experimenters  would  make  suggestions,  as 
necessary.  Next,  three  sets  of  medium  diff Iculty  simulator  tasks  were 
accompanied  by  an  auditory  interference  task,  a  visual  interference  task, 
and  a  computation  task.  The  auditory  interference  task  required  subjects  to 
say  whether  the  first  of  two  tape-recorded  tones  was  higher  or  lower  than  the 
second.  The  tone  pairs  were  presented  at  a  rate  of  12  per  minute.  The  visual 
interference  task  required  subjects  to  keep  track  of  a  tic-tac-toe  game  and 
to  say  whether  X's  or  0's  won  or  draw.  The  tape-recorded  task  was  presented  by 
verbal  instructions  such  as  "X  in  square  7...0  In  square  2",  etc  at  a  rate  of 
about  10  per  minute.  The  computation  task  required  subjects  to  cowpute  time  to 
the  next  station  given  air  speed  and  distance  on  a  3  x  5  card.  Subjects  were 
required  to  write  the  time  of  station  passage  and  the  estimate  of  Lime  to 


the  next  station  in  fractions  of  hours  and  in  seconds.  The  last  set  of 
three  tasks  repeated  the  sequence  of  interference  tasks  used  in  the  second 
set  of  three  with  the  only  difference  being  that  the  difficult  simulator 
conditions  were  used. 

Two  of  the  pilot  trainees  were  beginners  with  less  than  10  hours. 
They  both  had  a  great  deal  of  trouble  with  the  moderate  difficulty  simu¬ 
lator  task,  si'  we  did  not  give  them  the  difficult  simulator  task.  This 
hurt  our  correlations  because  they  would  certainly  have  performed  poorly 
had  we  put  them  through  that  ordeal. 

Because  of  the  small  number  of  subjects  remaining  at  the  end  of  this 
primary  simulation  task,  we  decided  against  running  more  advanced  simula¬ 
tion  tasks  designed  to  discriminate  among  more  advanced  pilots.  These 
tasks  include  difficult  versions  of  holding  patterns,  approach  patterns 
and  a  VOR  orientation  task. 


Results 

Our  primary  interest  is  in  whether  the  simulator  provides  a  valid 
measure  of  piloting  skill.  We  decided  on  two  achievement  measures  and  one 
judgemental  measure,  although  the  later  (ratings  by  the  trainee's  instructor's) 
is  not  yet  available.  The  achievement  measures  are  number  of  training  hours 
(range  between  3  and  150)  and  rating  achieved  (range  between  pre-solo  and 
Instrument  instructor).  Rating  achieved  has  not  proven  valuable  because 
surviving  data  are  too  bunched  up  in  the  middle  of  that  small  range  of 
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Data  on  simulator  performance  to  be  reported  here  are  rank  orderings 
of  traces  of  "ground  track"  relative  to  desired  course.  Judgements  were 
subjective  based  primarily  on  accuracy  and  smoothness.  Two  judges  (one  with 
some  flying  experience,  one  without)  ranked  the  traces:  the  average  correla¬ 
tion  between  raters  for  the  different  conditions  was  .92.  The  two  ranks 
were  averaged  and  the  average  scores  were  ranked.  Example  tracing  are  shown 
in  Figure  II.  (See  page  following). 

Recall  that  there  were  two  levels  of  simulation  task  difficulty  during 
which  the  calculation  task  was  simultaneously  administered.  Correlation  between 
training  hours  and  simulator  performance  with  the  simultaneous  calculation  task 
was  -.54  for  the  easier  task  and  -.68  for  the  difficult  task.  Since  one  is  the 
highest  rank,  the  negative  correlations  with  hours  of  training  are  as  expected. 
These  correlations  would  be  somewhat  higher  if  we  have  low  scores  for  the  two 
pre-solo  pilots  who  could  not  complete  the  difficult  simulator  conditions. 

A  few  words  of  caution  are  in  order  about  interpretation  of  these  high 
correlations.  Much  data  were  lost  due  to  computer  failures  and  subject  at¬ 
trition  so  that  the  numbers  of  observations  are  small  (14  for  the  easier  tasks, 
11  for  the  difficult  task,  not  counting  the  two  beginning  pilots  who  were 
unable  to  complete  it).  Although  both  of  the  scores  are  significant  (we 
are  using  a  t-test  with  p  <  .05  throughout)  the  actual  values  of  the  corre¬ 
lations  should  not  be  taken  as  good  estimates  of  accounted  for  variance. 
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Wg.  11  Tracings  of  desired  courses  (solid  line)  and  actual  performance  (dotted 
line  ground  track).  The  better  tracing  is  from  an  Instrument  Instructor 
with  1000  flying  hours  including  140  flight  training  hours.  The  other 
tracing  is  from  a  private  pilot  with  60  hours  of  flying  time  Including 
40  hours  of  training.  Tracings  are  from  the  most  difficult  simulator 
condition  and  computations  were  required. 
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We  expected  high  correl.it  Ions  between  the  two  calculation  tasks  which 
differed  only  in  difficulty  of  the  simulator  task.  The  significant  corre¬ 
lation  was  .83. 

Since  we  believed  that  direction  maintenance  without  task  complexity 
would  not  be  an  important  predictor,  we  expected  low  correlations  between  the 
simulator  task  alone  and  the  combined  simulator  and  calculation  tasks.  For  the 
moderate  difficulty  condition  the  non-significant  correlation  was  .14  for 
the  difficult  conditions,  however,  the  significant  correlation  was  .73.  Thus 
it  appears  that  the  same  subjects  are  doing  well  on  the  difficult  conditions 
regardless  of  whether  or  not  there  Is  a  calculation  task  for  added  complexity. 

Wc  hoped  to  gain  some  insights  about  coding  processes  through  the  use 
of  the  interference  conditions  but  the  small  numbers  of  data  points  and 
possible  insensitivity  of  the  rank  order  data  make  it  unclear  whether  visual 
interference  was  more  harmful  to  experienced  pilots  than  to  non-experer iciued 
pilots.  There  is  not  a  meaningful  pattern  of  significant  differences  and  what 
data  there  are  do  not  suggest  that  visual  interference  was  more  important  to 
experienced  pilots  than  auditory  interference.  We  believe  that  the  absolute 
scores  derived  from  the  various  statistics  which  we  are  recovering  from  the 
data  will  be  more  sensitive  to  this  matter  than  ranks.  The  advanced  pilots 
were  so  much  better  than  the  poorer  pilots  that  the  differences  caused  by 
interference  might  not  have  been  great  enough  to  bo  picked  up  by  changes  in 
ranks . 

Discussion 

We  believe  that  we  have  tentative  validation  of  the  simulation  task; 
the  simulation  seems  to  be  sensitive  to  some  of  the  important  aspects  of  pilot 
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training-  Naturally  we  would  like  to  have  a  more  complete  validation  but  due 
to  lack  of  availability  of  highly  skilled  pilots  we  believe  that  further 
development  is  best  done  at  another  location. 

It  setms  likely  to  us  that  discovering  good  pilot  ability  Is  best  done 
with  techniques  using  information  overload.  Hany  pilot  tasks  are  not  demanding 
in  most  situations.  Turning  onto  a  final  approach  path  smoothly*  for  example, 
can  be  done  relatively  easily  by  most  fairly  experienced  pilots  when  there  is 
nothing  else  to  do.  On  instruments,  in  turbulence  with  one  or  two  radios  out, 
and  going  twice  normal  speed  might  make  this  task  more  challenging  and  dis¬ 
criminating  of  those  who  can  handle  bad  situations  while  piloting.  This  is 
the  sort  of  challenge  that  can  be  sat  up  by  a  simulation.  Another  measurement 
issue  is  repeatability.  One  does  not  have  to  wait  for  a  windy  day  to  check 
out  cross  wind  landings  or  gusty  wind  landings  and  such  situations  are  exactly 
repeatable  for  measurement  considerations.  Finally,  there  is  the  issue  of 
measurement  recording  accuracy.  One  need  not  judge  distance  fiom  desired 
course  or  smoothness  and  accuracy  of  turns  by  eye.  Ground  track  can  be  plotted 
exactly  along  with  other  statistics  such  as  numbers  of  course  corrections. 

Thus  the  flexible  simulator  allows  more  discr  imin*Te,  repeatable,  accurate 
measurement. 

Our  current  goal  is  to  use  the  simulator  on  our  readily  available  sub¬ 
jects  to  see  whether  we  can  predict  simulator  performance  (during  training  and 
at  assymptotic  levels)  by  performance  on  other  cognitive  abilities.  The  tasks 
which  we  have  ready  to  go  measure  ability  to  represent  visual  material,  the 
ability  to  set  priorities,  attentions!  flexibility,  and  planning  ability. 
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